Skip to content

[fix](function) Preserve trailing zero bytes in string distances - #66236

Open
Mryange wants to merge 1 commit into
apache:masterfrom
Mryange:remove-string-distance-tail-trim
Open

[fix](function) Preserve trailing zero bytes in string distances#66236
Mryange wants to merge 1 commit into
apache:masterfrom
Mryange:remove-string-distance-tail-trim

Conversation

@Mryange

@Mryange Mryange commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary: String distance functions trimmed trailing zero bytes from every input during execution. CHAR padding is already removed at the storage read boundary, so this work was redundant for CHAR values and incorrectly discarded legitimate trailing zero bytes from STRING and VARCHAR values. Use the complete ColumnString values for Hamming, Levenshtein, and Damerau-Levenshtein distance calculations.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: apache#63291

Problem Summary: String distance functions trimmed trailing zero bytes from every input during execution. CHAR padding is already removed at the storage read boundary, so this work was redundant for CHAR values and incorrectly discarded legitimate trailing zero bytes from STRING and VARCHAR values. Use the complete ColumnString values for Hamming, Levenshtein, and Damerau-Levenshtein distance calculations.

### Release note

Fix string distance functions to preserve trailing zero bytes.

### Check List (For Author)

- Test: Not run; awaiting author review before compilation and test validation
- Behavior changed: Yes; trailing zero bytes now participate in string distance calculations
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Mryange

Mryange commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review: request changes

Two blocking issues are in the inline comments: a NULL row can now abort Damerau-Levenshtein before its null map is reapplied, and no regression reaches the BE paths changed by this patch. Please also update the PR metadata: trailing-NUL distance results deliberately change, so Release note: None, the unchecked Behavior changed item, and the empty test checklist do not accurately describe the PR.

Critical checkpoints:

  • Goal and proof: non-null inputs consistently retain real trailing NUL bytes across Hamming, Levenshtein, and Damerau-Levenshtein, but mixed nullable blocks can fail before NULL is reapplied and no base-distinguishing test proves the intended behavior.
  • Scope and clarity: the production change is small and focused on the two affected BE implementations.
  • Concurrency and lifecycle: no shared mutable state, locks, threads, static initialization dependency, or special resource lifetime is introduced.
  • Error, nullability, and parallel paths: Hamming's custom nullable paths skip NULL rows, but the shared Levenshtein/Damerau wrapper evaluates arbitrary nested NULL-row payloads and can surface a matrix-cap error; const/vector routes, aliases, and FE folding otherwise use complete values consistently.
  • Configuration and compatibility: no configuration, protocol, serialized format, or FE-BE variable changes are introduced. Production CHAR storage read paths remove padding before expression evaluation, while STRING/VARCHAR lengths remain intact.
  • Tests and results: blocking gap; existing tests cover ordinary branch shapes but never the changed trailing-NUL behavior or the mixed nullable error path, and this PR changes no generated result file.
  • Performance and memory: removing repeated tail scans is favorable for non-null values and adds no allocation site, but arbitrary NULL-row payloads must not trigger avoidable distance-matrix work.
  • Observability, persistence, transactions, and data writes: not applicable to this expression-only change.
  • User focus: no additional focus was provided; the whole PR was reviewed.

Review was static as required by the action prompt; no build or test command was run.

const auto previous_offset = i == 0 ? 0 : offsets[i - 1];
return StringRef(data.data() + previous_offset, offsets[i] - previous_offset)
.trim_tail_padding_zero();
return StringRef(data.data() + previous_offset, offsets[i] - previous_offset);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Do not evaluate retained payloads for NULL rows

This function family uses the default nullable wrapper with need_replace_null_data_to_default() == false, so mixed nullable blocks are evaluated on arbitrary nested values before the null map is reapplied. NULLIF(s, k) retains s under its NULL row; if one row has s = k = 4096 NUL bytes and another row is non-NULL, the base trims that nested row to empty, but this line keeps all 4096 bytes. damerau_levenshtein_distance then rejects (4096 + 2)^2 = 16,793,604 cells above the 16,777,216 cap and aborts the query even though that row's result must be NULL. Please replace or skip null-row nested data before running the distance implementation and add mixed null/non-null coverage.

RETURN_IF_ERROR(scalar_vector(left_str_col->get_data_at(0).trim_tail_padding_zero(),
*right_str_col, res_data));
RETURN_IF_ERROR(
scalar_vector(left_str_col->get_data_at(0), *right_str_col, res_data));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Add a regression that reaches the patched BE paths

These removals are the entire behavior change, but the changed-file list contains no test/output file and the existing distance suite never constructs a trailing NUL. Literal-only cases are insufficient because Nereids StringArithmetic already counts U+0000, so they can pass on the base revision without executing these BE branches. Please add generated regression coverage that fails on the base and passes here, using table-backed binary values to exercise runtime vector/nullable and const-vector paths for Hamming, Levenshtein, and Damerau-Levenshtein (for example a\0 vs a and a\0 vs aX), plus a CHAR case proving storage padding stays invisible.

@Mryange

Mryange commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

/review 当前这个pr只是去掉了trim_tail_padding_zero,如果你发现了一些问题,可能是之前的遗留问题,当前pr不考虑这个

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated follow-up review: changes still requested

Round 1 converged with two normal full-review subagents and a separate risk-focused subagent. All three returned NO_NEW_VALUABLE_FINDINGS: no new inline issue survives validation or duplicate suppression. The two existing P1 threads are still unresolved, non-outdated, and anchored to the current head:

  • A mixed nullable block can evaluate a retained trailing-NUL nested payload before the null map is reapplied, so a semantically NULL row can abort Damerau-Levenshtein at its matrix cap.
  • No regression currently reaches the changed BE paths with trailing-NUL data or proves that CHAR storage padding stays invisible.

Critical checkpoints:

  • Goal and proof: the non-NULL execution paths consistently preserve legitimate trailing NUL bytes for Hamming, Levenshtein, and Damerau-Levenshtein, but the nullable failure and missing base-distinguishing regression mean the goal is not safely proven end to end.
  • Scope and clarity: the production diff is small, focused, and limited to removing tail trimming in the two affected BE implementations.
  • Concurrency: no threads, shared mutable state, locks, atomics, or lock ordering are involved.
  • Lifecycle and initialization: no new static initialization dependency, ownership cycle, cache handle, or special resource lifetime is introduced.
  • Error handling, nullability, and parallel paths: Hamming skips NULL rows before evaluation and propagates failures; the shared Levenshtein/Damerau default nullable wrapper does not. Vector/vector, both constant orientations, ASCII/UTF-8 branches, aliases, and FE folding otherwise use complete values consistently.
  • Configuration and compatibility: no configuration, FE-BE variable, protocol, function symbol, or persisted/storage format changes are introduced. No rolling-upgrade compatibility branch is required for this scalar semantic correction. Current CHAR direct, dictionary, fallback, and statistics read paths strip physical padding before expression evaluation, while VARCHAR/STRING preserve trailing NUL.
  • Conditional logic: no new special condition is added; the existing Hamming null checks have concrete nullable-row paths and remain consistent across branches.
  • Tests and results: blocking gap. Existing tests cover ordinary dispatch shapes but contain no table-backed trailing NUL, no CHAR comparison for this behavior, and no mixed nullable retained-payload case; this PR changes no generated result file. Literal folding cannot prove the BE fix because Nereids already counts U+0000.
  • Performance and memory: removing repeated tail scans is beneficial on non-NULL inputs and adds no allocation site, but the nullable path can now perform avoidable quadratic work or fail the existing matrix limit on a result that must be NULL.
  • Observability: no new operational path needs logs or metrics.
  • Persistence, transactions, data writes, and crash recovery: not applicable to this expression-only change.
  • Other review state: the existing review-body request to align Release note: None, the unchecked behavior-change item, and the empty test checklist with the deliberate user-visible change remains applicable.
  • User focus: I treated unrelated legacy concerns as out of scope, as requested. The two retained blockers are directly tied to this trim removal: one is newly exposed by preserving the payload, and the other is the missing proof for the behavior this PR changes. No additional legacy issue is being raised.

This was a static review as required by the action prompt; no build or test command was run in this review environment.

@HappenLee HappenLee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@HappenLee

Copy link
Copy Markdown
Contributor

run buildall

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jul 30, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@Mryange

Mryange commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29259 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bd44cb1d9f7023e220685b00fd0eb5f0b41a802e, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17964	4001	3996	3996
q2	2302	320	198	198
q3	10533	1358	830	830
q4	4830	466	338	338
q5	8023	832	568	568
q6	197	176	138	138
q7	774	821	616	616
q8	10604	1642	1617	1617
q9	6639	4282	4266	4266
q10	7101	1703	1440	1440
q11	739	347	324	324
q12	889	575	465	465
q13	18636	3293	2713	2713
q14	271	267	244	244
q15	q16	789	777	703	703
q17	1051	903	998	903
q18	7428	5884	5651	5651
q19	1560	1335	993	993
q20	858	675	562	562
q21	5657	2535	2386	2386
q22	432	356	308	308
Total cold run time: 107277 ms
Total hot run time: 29259 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4434	4323	4307	4307
q2	293	331	223	223
q3	4561	4977	4410	4410
q4	2095	2190	1387	1387
q5	4404	4580	4451	4451
q6	230	189	141	141
q7	1898	1761	1524	1524
q8	2376	2041	2086	2041
q9	7681	7744	7676	7676
q10	4718	4626	4157	4157
q11	563	419	410	410
q12	767	763	557	557
q13	3311	3564	2956	2956
q14	268	279	259	259
q15	q16	707	710	628	628
q17	1302	1285	1247	1247
q18	7635	7125	7418	7125
q19	1153	1112	1111	1111
q20	2241	2245	2020	2020
q21	5525	5001	4712	4712
q22	550	531	454	454
Total cold run time: 56712 ms
Total hot run time: 51796 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 177950 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bd44cb1d9f7023e220685b00fd0eb5f0b41a802e, data reload: false

query5	4340	639	477	477
query6	502	225	209	209
query7	4946	591	314	314
query8	450	196	175	175
query9	8807	4053	4045	4045
query10	487	371	297	297
query11	5906	2356	2150	2150
query12	162	104	103	103
query13	1267	629	408	408
query14	6191	5243	4874	4874
query14_1	4246	4205	4188	4188
query15	215	197	176	176
query16	3475	503	422	422
query17	1129	727	541	541
query18	2240	480	332	332
query19	216	189	146	146
query20	116	104	104	104
query21	716	162	139	139
query22	13806	13505	13486	13486
query23	17508	16779	16547	16547
query23_1	16567	16402	16229	16229
query24	8119	1923	1380	1380
query24_1	1351	1383	1353	1353
query25	600	520	421	421
query26	1583	373	229	229
query27	2634	627	391	391
query28	4454	2063	1997	1997
query29	1111	610	466	466
query30	462	264	221	221
query31	1128	1091	970	970
query32	98	57	56	56
query33	504	329	248	248
query34	1204	1113	644	644
query35	779	774	658	658
query36	987	1008	871	871
query37	151	98	86	86
query38	1875	1739	1626	1626
query39	896	908	841	841
query39_1	823	856	844	844
query40	327	167	138	138
query41	66	61	61	61
query42	88	89	91	89
query43	318	325	273	273
query44	1478	770	769	769
query45	191	176	175	175
query46	1158	1218	734	734
query47	2048	2102	2014	2014
query48	411	424	303	303
query49	580	424	305	305
query50	1151	437	347	347
query51	10915	10838	10329	10329
query52	87	87	74	74
query53	261	283	204	204
query54	279	235	220	220
query55	75	70	63	63
query56	308	300	291	291
query57	1477	1291	1174	1174
query58	283	253	292	253
query59	1576	1664	1400	1400
query60	327	269	255	255
query61	154	149	148	148
query62	561	498	429	429
query63	253	202	202	202
query64	2451	1035	836	836
query65	4738	4695	4699	4695
query66	1752	487	373	373
query67	29518	29388	29174	29174
query68	3263	1586	1029	1029
query69	515	313	277	277
query70	935	823	844	823
query71	377	345	320	320
query72	3492	2680	2524	2524
query73	868	775	430	430
query74	5119	4943	4739	4739
query75	2642	2547	2196	2196
query76	1922	1223	860	860
query77	393	404	283	283
query78	11878	11993	11430	11430
query79	1496	1183	773	773
query80	1274	599	501	501
query81	619	338	287	287
query82	678	203	118	118
query83	375	327	302	302
query84	383	158	135	135
query85	1071	632	540	540
query86	469	243	241	241
query87	1847	1820	1732	1732
query88	3791	2788	2803	2788
query89	438	371	341	341
query90	1958	199	198	198
query91	203	188	161	161
query92	61	58	57	57
query93	1681	1567	948	948
query94	750	345	326	326
query95	814	523	548	523
query96	1110	804	355	355
query97	2658	2641	2475	2475
query98	221	206	197	197
query99	1151	1120	973	973
Total cold run time: 270958 ms
Total hot run time: 177950 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.02 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bd44cb1d9f7023e220685b00fd0eb5f0b41a802e, data reload: false

query1	0.01	0.01	0.01
query2	0.42	0.04	0.04
query3	0.76	0.14	0.13
query4	2.07	0.14	0.14
query5	0.24	0.21	0.22
query6	1.93	0.82	0.81
query7	0.04	0.00	0.01
query8	0.06	0.04	0.04
query9	0.71	0.31	0.32
query10	0.56	0.54	0.54
query11	0.37	0.14	0.14
query12	0.26	0.14	0.14
query13	0.47	0.48	0.48
query14	1.03	1.03	1.03
query15	0.63	0.61	0.60
query16	0.33	0.35	0.31
query17	1.08	1.15	1.10
query18	0.25	0.21	0.22
query19	2.35	1.98	2.04
query20	0.02	0.01	0.02
query21	16.36	0.18	0.14
query22	4.81	0.06	0.06
query23	16.94	0.30	0.12
query24	18.14	0.41	0.33
query25	0.11	0.06	0.05
query26	0.74	0.20	0.16
query27	0.04	0.04	0.03
query28	3.61	0.96	0.56
query29	12.73	4.24	3.32
query30	0.28	0.15	0.15
query31	2.87	0.62	0.33
query32	3.43	0.60	0.49
query33	3.14	3.20	3.18
query34	15.52	4.25	3.53
query35	3.50	3.56	3.57
query36	0.55	0.43	0.42
query37	0.20	0.06	0.07
query38	0.05	0.04	0.03
query39	0.17	0.03	0.03
query40	0.27	0.17	0.15
query41	0.36	0.03	0.03
query42	0.20	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 117.65 s
Total hot run time: 25.02 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 0.00% (0/20) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 59.04% (25556/43285)
Line Coverage 43.12% (256235/594169)
Region Coverage 38.80% (202953/523032)
Branch Coverage 40.15% (92566/230541)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.1.x

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants